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A PROCEDURE FOR ESTIMTING THE UNIQUI CONTRIBUTION OF EACH COMPONENT Of A 
COOTOSITE TESTi UNIQUENESS ANALYSIS Of TEST 500 



ABSTRACT 



A procedure for estimating the degres to which a lubtcst uniquely contributes to 
total tfist performanee is presented and discussed. Uniqueness analysis may be appro- 
priately applied to any composite measurement instrument such as a multipart test or 
a multitest battery to assess the unique contribution of each component to the total test. 
The preaentation in this report is In terms of the applicability of these procedures to 
Test 500 of the Prof essional and Administrative Career EKamlnation* Uniqueness aaalyses 
were conducted on each series of Test 500 administered compatitively in IT 75 • The result 
for each jpf the five subtests, showing the degree to which subtest variance can be attrib- 
uted to error, overlap, and unique apureeSs and Indicating the potential unique validity 
of each subtest are presented by test series. . 



PREFACE 



The purpose of this Technical Memorandum is to pra^ent and demonetratfe a procedure 
for determining the proportion of unique and reliable variance in m aubteet that tm being 
used in a linear composite test or in a test battery. It is designed primarily for 
payehologlats of the U. S. Civil Service Commission for use in teat development. The 
uniqueness analyals procedure was applied to th^ four series of Teet 500 of the Professional 
and Administrative Career Exmlnation administered in FY 75 and serves as part of the doeu- 
mentation of thia test. 
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A PROCmUEE FOR EBTmkTmG THI UNIQUE CONTRIBUTION 
OF EACH COMPONENT OF A COMPOS ITB *^ESTr/ 
UNIQUENESS AKALYSIS OF TEST 500 



Test 500 of the Prof esslonal and Admin- 
istrative Career Examination (PACE) is a 
written test of five abilities shoTO by job 
analytic research to be necessary for sue- 
eessf ul perf oraaiice In a amaber of federal 
occupatisns* The five subtest aeores are 
differentially weighted for sIk different 
patterns of oeeupatlons- A linear composite 
of the weighted ability scores is used to 
determine an applleant's examination rating 
for each pattern (Wlsg, 1974) • 

Underlytog the composite nature of Test 
500 and the differential scering procedure 
is the assumption that each of the five 
ability subtests makes a unique centributlon 
to the deseriptlve and predictive efficleney 
of the total test. It is assumed that per-- 
formance on each subtest relies to some 
degree on abilities not tapped by the other 
subtests; eaeh subtest score Indieates soma 
unique aspect of an applicant's abilities. 
Therefore > Inforaation obtained from one 
subtest score does not replicate information 
from the ot^er subtest seores In describing 
applicants' abilities and ranking them ac- 
eordlngly* Furtheraorep since eaeh of 'the 
subtests has been designed to measure an 
ability shorn to be necessary for sueeessful 
Job perforaaace* a subtest score which 
uniquely contributes to total test perform- 
aace also adds to the validity of the total 
test against criteria of Job performance. 

This Technical Memorandum provides test 
users and psychologists with a comprehensive 
procedure for assessing the unique contri- 
bution of components of a composite measure^ 
sent Inst rtment, Basle to the understandlGg 
md derivation of this procedure is the con- 
cept of variance In test scores | i,e.j the 
sources of test variance and methods for deter- 
mining how much of the varirace in a composite 
test is attributable to each co^oneat source 
should be understood* These concepts are dis- 
cussed in detail and statistical methods for 
their quantification are given » 

Techniques for assessing the unique 
contribution of each subtest of a composite 
test were applied to each of the four series 
of Test 500 administered competitively in 
nr 75 (Series 110, 120, 130, and 140) . ^e 



determination of the variance components for 
each subtaat are presented and discussed 
as wall as the results of the unlqumess 
analyses of each subtest of each series, 

COMPONENTS OF VARIANCE IN TEST SCORES 

When tests are reliable i variance in 
scores results for the most part from indi- 
vidual differences among the competitors 
taking the test (Guilford S Pruchter, 1973, 
p. 68)* Not all the variation in scores is 
due to such individual differences however, 
because a single test score is rarely a 
perfect indication of a person's true score 
on the variable in question* In the classi- 
cal test theory model (Lord ^ Novick, 1968 , 
p* 31), this discrepancy between true score 
and observed score is attributed to error 
arising from i^erfect measuring instriments 
(errors of measurement) , 

It is important for test users to know 
how much of the variance in observed test 
scores is attributable to variance in tadi- 
viduals' true scores. For test scores to 
differentiate ^Mng competitors, it is neces- 
sary that there be variability in their true 
scores. The true score variance is reflected 
in the observed score variance. If the 
observed score variance arises from error 
rather than from real differences among the 
competitors however, their relative place- 
ment along the range of scores is based on 
chanae and gives no meaningful informtion 
about their abilities. 

Procedures for estimating the reliability 
of a test deal with the issue of true and 
error varianca in test scores* Although 
there are a igtm^er of statistical procedures 
for estlmttog rallabllity^ aach arising from 
a different conception of the gources and 
types of errors of measurenent md therefore 
leading to different interpretationsV relia- 
bility can be defined as the proportion of : 
obse^ed score variance of a set of measure* 
ment s that is true vas?iance (Guilford i 
Fruchter, 1973, p, 397)* ^us, the relia- 
bility coefficient of test indicates the 
proportion of the variance^of observed scorea 
that is attributable to true differences 
Mong eoa^etitors. 



The cbncept of test score variance be"- 
comes more Gomplex when dealing with a 
composite measurament instrument. In this 
case, the total test variance is a function 
of the variances and covarlaneea of the 
subtests » Nonzero covariancea mong the 
aubteats indierte that these aubteata share 
a true score variance component. (See 
Martin, 1975 for a more complete treatment 
of this topic.) The total variance of each 
subtest can be conaldered as the sum of 
variance from three apurcesi (1) that 
common to more than one of the subtests ; 
(2) that unique to the sub teat I and (3) 
that due to errors of measurement. The 
third component has already been diacuased. 
Only the reliable subtest variance can be 
meanlngf ullypartitioned into coimnon and 
unique components. 

Fa^titioniYig VmimQB 

Correlation analysis serves to part lo- 
tion the variance of a particular variable 
into two Independent soureee^^one aoutce 
which can be eKplained by another variable 
and a second source which cannot be eKplained 
by that variable. In the multivariate caae^ 
^-v correlational analysis partitions the varl-- 
ance of one variable into sources that can 
and cannot be accounted for by comblnafelona 
of other variables. 

In a composite tests intercorrelationa 
among pai%"a of subteats indicate the degree 
to whieh variance in one subtest score can 
be attributed to variance in the ' other sub-- 
test* To determine how much of the vari^ 
ation in one subtest score In a composite 
test is due to variation in all the other 
subtests* multiple correlation techniques 
must be used (McNemar, 196&» Chap, 11)* If 
the other subteats each correlate Eero; with 
the subtest in questions then ^there is no 
coimnon variance. This situation is rare 
in the area of psychological testings how=- 
ever s because mental abilities tend to be 
. interrelated t Generally subtests do corre^ 
late with each other to some extents making 
the degree of their uaiqueneas difficult to 
determine. The procedures described In this 
paper apply in such cases. 
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As noted aboves multiple correlatlori 
techniques are appropriate for deteTOlnlng 
the degree to which one subtest score can 
be predicted from an optimally weighted 
combination of the rest of the cempQnents 
of the test. The multiple R therefore 
represents the masstMun degree to which one 
test eompenent Is correlated with the 
remaining co^onenta* The squared value of 
this statlatlc, the coefficient of multiple 
determination (R^) , Indicates the proportion 
of variance in the one subtest that ia ^ 
dependent upons associated with, or pre-- 
dieted by the other subtests- Since the 
scores for the other subteats are optimally 
weighted s R^ indicates the m^lmum posaible 
proportion of common variance. 

Since subtest variance has been defined 
as the sum of co^on. uniques ^nd error 
var lance i the unique var iance is that p r o - 
portion of the variance remaloing after the 
proportions of variance attributable to 
comon and error variance have been deter- 
mined* Uniqueness Malysis consists of a 
procedure for estlMting the proportion of 
variance coMon among subtests and subtract-^ 
Ing this from the reliable (non--error) 
variance of the subtest under study. A 
formal presentation of this procedure was 
given by Flanagan (1959) . 

UNIQUENESS ANALYSIS 

Flanaian's uniqueness coefficient 
repreaents the proportion of a subtest's 
total variance that is both unique (l.e,, 
free of overlap with variance measured by; 
any other subtest in the co^oslte) and not 
attributable to chance. It is defined by the 
formula I 

^cc 



where U^ ^ the uniqueness coefficient for 
^ variable 1 

til ^ reliability coefficient 
for variable 1 



the multiple cdttelmtlon 
(croas-valldatad) of variable i 
with the optimally weighted 
conq^osite (e) of the rest of 
the tests 

the reliability of the weighted 
coi^oelte of the independent 
varlablea (rest of the subtests) 



Equation 1 calls for a croes^-valldated 
multiple correlatloii coefficient CR|e) . The 
proeedure for optimally weighting aubtests 
In calculating the multiple R capitalizes 
on chance relationahlpa In the particular 
aample from which the scores were obtained, 
Iti maximizing the multipla correlationi 
these procedures take advantage of any cor-- 
related errors or apeciflc variations In the 
particular aampla studied giving apuriously 
high estiMtes of the correlations Mong the 
Subtests and of the multiple correlation. 
The degree to which chanea affects the multi- 
ple correlation depends" Inversely on the 
slza of the sample studied and^ where 
applicable * on the number of variables from 
which a smaller variable set is to be 
selected. As a result of such sampling 
fluctuations, applying the weight determined 
from one sample to predict the aame cri= 
terlon in a new sample will result In a 
lower multiple correlation between predic- 
tors and criterion In the new sample . As 
sample size Increases * sampling errors 
among correlatidna decreases and the multi- 
ple R is less affected by capitalisation on 
chance p Tliis tendancy for the multiple 
correlation to decrease as the sample size 
grows larger is called shrinkage; 

in order to obtain values of the 
multiple correlation coefficient which are 
stable and generallzable across samples i the 
weights should be cross-validated* This 
procedure Involves the application of 
weights or regression equations found in one 
sample to predictions in another smple to 
check on their general applicability and 
amount of shrinkage in the obtained multiple 
R. 



Another method for cotrecting tofiated 
multiple correlations involves the applica- 
tion of a shrinkage formula such as that 
developed by Wier^ (1939) • The mmvvy 
shrtal^ge f ormulal results in m estimate 
of the multiple R for an infinitely large 
aample from the value obtained on a sample 
of specified size. Wien there is pre- 
selection among variables this shrinkage 
f omula may not fully correct an inflated 
multiple R, but when i:here-ls _po preselec- 
tion aDtorig variables (all sampiing error Is 
due to selection of persons, not of varia- 
bles) j the formula gives an imblased esti-- 
mate of the multiple correlation md is an 
appropriate alternative to cross^-validatlon* 
The corrected correlation coef f IciMt repre-^ 
STOts the maximum degree to which the com- 
posite is related to the refflaining subtest 
(variable i)| the variance due to sampling 
errors is eliminated from the result, 

Flanagan's procedure for estimating the 
proportion of variance coTOon to one subtest 
and the combination of the other subtests 
corrects the estlMte given by the multiple 
coefficient of determination (R^) for the 
unreliability in the independent variables 
(in this case^ the other subtests)* The 
estimte is corrected for attenuation due to 
errors of measurement in the composite 
instrment. The resulting squared correla- 
tion Indicates the mascimiim proportion of 
conmon variance possible if the composite 
variables had all been measured t^th perfect 
reliability. The 

^ce 

term In Equation 1 therefore represents the 
proportion of true score variance in the 
composite measure comon to the observed 
score variance in variable i. The correc- 
tion for attenuation is applied in this case 
because what is of interest is the degree of 
overlap between true performance on the 
co^oslte of other subtests and actual per- 
fo^nce on the remaining subtest. 



!^e Wherry shrinkage formula iss 



where N • nmber o£ cases 

n ■ ni^er of independent varlableB 
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The acatlatic needed to make the correc- 
tion for attenuation, r^^, represents the 
reliability of the weighted cgmposlte of the 
rest of the aubtesti^v Procedures for obtain- 
Ing such reliability estimates are given in 
Hartin (1975), which also discussed differ- 
ent conceptual interpretations of reliablll^ 
ty and the three Mst comon procedures for 
obtaining reliability coefficients. Unique- 
neas analyaia does not require use of a 
specific type of subtest reliability . 
eattaate; the teat analyat ahould be aware 
of the different assumptiona underlying the 
various reliability techniques and be 
careful to apply procedures appropriate for 
the measurement instniment being studied to 
obtain reliability eatimates used in unique- 
neaa analysis » 

The reliability c^^ficients r^jLi is 
the proportion of true ac%^ or non-error 
variance in variable 1, The^^^^ortion of 
varimce due to errors of measurment is 
l-t±±* Subtracting the estimate of over- 
lapping variance t 




from the proportion of non^error variance 
indicates the proportion of variance in 
variable i that ia both unique and not 
attributable to chmce. 



Equation 1 therefore partials out both 
the error variance in variable i* the 
particular subteat being studied (by using 
the reliability eatisate t±±) t and the 
error variance in the estimate of coinnos 
variance (by correcting the value for 
both sampling errors and errors of measure^ 
ment in the composite) , Flanagan (1964) 
notes I 'and it is apparent from Equation !» 
that underestimation of the subtest relia^ 
bilities can result in negative values of 
U^t This effect la a result not only of a 
smaller r£^ value from which to subtract the 
estimate of overlapping variance ^ but also 
of a spuriously high value when 
corrected for attenuation by an Under- 
estimate of the reliability of the composite* 



Interposition of ihm UnicpmteBa 
CoeffiaiBnt 

According to Ilanagan (1964) i 

t».a test is making a unique 
contribution to the battery if 
the multiple correlation (pref- 
erably croaa-validated) between 
the test and an optimally 
weighted composite of all the 
rest of the testa is signifi- 
cantly less than the maximtwi =^ 
correlation consiatant with 
the reliability of the vari- 
ables correlated, (p. 2-59), 

"Significantly" Is not defined in tema 
of statistical probability, but ±a _ 
practical terms* As the value of Hj^, 
consistent with the reliability of the 
weighted compoalte, r^^, approaches the 
reliability of variable i, r^i, the 
value of Uj decreases. As noted above, 
u| may even be negative If underesti^ 
mates of reliability are used^ although 
theoretically zero is the lower bound 
(lie,* the subtest la question Is making 
no unique contribution to teat variance), 
toy positive value of u| is therefore 
directly imterpretable as the proportion 
of unique varimce in that subteat 
(variable i)# 

Flanagan discussed the aignificance 
of u| values in practical, operational 
tema* Any value of u| greater than 
lero Jjidicates that the subtest is making 
a unique contribution to the total test 
acoret Practically^ the issue of the 
degree to which a subtest uniquely con- 
tributes to the total test acore is 
related to the predictive value of the 
total test. If the subteat is corre^ 
lated with the criterion, any additional 
information in the test acore (unique 
variance) related to the criterion acts 
to increase the validity of the total 
test* The degree to ^ich each subtest 
potentially contributes to the total test 
validity is given by u|* Since u| 
represents the proportion of reliable 



variance unique to the subtest, it also can 
be interpreted as the maKimum degree to which 
the subtest -a unique variance can account 
for variance in a pure criterion measure of 
its unique function. Since the square of 
the correlation coefficient batween two 
variables indicates the proportion of their 
coOTon variance, u| can be interpreted as a 
squarad correlation coefficient* Therefore, 
Ui is an aatiMte of the maximum possible 
partial validity coefficient of a subtest 
with a pure criterion measure of the test's 
unique function when the teat ia used In the 
given teat battery or a linear compoaite* 

UNIQUENESS ANALYSIS OF TEST 500 

Flanagan's procedure for uniqueness 
analyaia was applied to the four aeries of 
Teat 500 adminiatered competitively in FY 
75* Each, aeries conslsta of five subteats 
designed to measure different abilities. 
Four of the abilities are measured by two 
types of test items, and one ability is 
measured by only one item type. Although 
alternate-forms reliability eatimates indi- 
cate that the aubtasta In different aeries 
measure the same abilities (tortlnj 1975)* 
saparate uniquenesa analysas of each serlea 
were carried out in thia initial effort. 

Internal conalstency (KR-20) relia- 
bility coefficients were used in tha analysis 
under the aaaumption that arrore of measure- 
ment reaultad from lack of internal consis- 
tency in item aampllng. These reliability 
coafficiants were also uaed in eatlmating 
the reilabllity of the optimally weighted 
composite of the four subteats, 

r Multiple correlation coefficients 

obtained on large samples (e*g. greater than 
IjOOO cases) are relativaly unbiased popu- 
lation estimates, particularly when the 
number of variables in the composite is 
small. It ia obvious from the formula that 
ill such situations the l^erry correction of 
the obtained multlpla correlation coeffl-* 
cient is generally beyond the second decimal 
place* The values obtained from the 
Test 500 data are unbiased to the fifth 
diclmal place, making correction for 
shrinkaga umecessary. It is doubtful that 
any large error resulted from using tha 
^corrected values since the smple 
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sizes were so large and far exceeded the 
number of variables and the multiple corre- 
lations coefficients ware high enough to 
represent nonchance relationships. 

Tables 1 through 4 give the reaulta of 
the uniqueness analyses by test serlea. 
The proportion of variation attributable 
to error, overlapping, and unique variance 
are given. Also presented are the potential 
unique validity coefficients for each test 
part. Since the five subteats of Teat 500 
ware designed to meas^e job-related abili- 
ties, thla validity estimate is meaningful 
because It ia certain that the unique 
ability measured by each subtaac woiild be 
part of any criterion of successful Job 
performance. 
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TABLE 1 



Uniqueness Analysis - Test 500, Series 110 



Test Part 



-'A 
C 
D 
E 
F 



PropDrt.Lon of 
Error Varianae 

1-^11* 



,146 
.264 
.224 
.169 
.265 



Multiple 
Correlation 



.749 

.739 
»611 
.767 
.622 



Proportion of 

Overlapping 

Variance 

ic 



• 652 
•.604 

• 425 
.645 

• 433 



Proportion of 

Unique 

Variance 

4 ' ' 



.202 

• 132 

• 351 
,186 
.303 



Potential 
Unique 
Validity 
U^ 



.449 
.363 

• 592 
.431 

• 550 



TABLE 2 

Unlquaness Analysis - Test 500, Series 120 



Proportion of 

0^/erlapping- Proportion of Potential^ 

. Proportion of Multiple ^ ^ ^^^^^ V Unique Unique ^ 

Error Variance Correlation rL Variance Validltv 

Teat Part l-r^i^ R^^ u| u. 



*See Equaelon 1 for explanation of statistical ti 



A .151 .740 .645 .204 452 

C -aOl .726 .583 .116 .341 

D -195 .621 .437 .369 .607 

E -183 .757 .627 .191 .437 

P •253_,_. .617 .425 .323 .568 
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Uniqueaaai^taalysis Test 500, Serlas 140 



Taat Part 



Proportion of 
Error Variance 



1-r 



Hultipla 
Corralaclon 



Froporatlon of 

Overlapping 

Variance 



Proportion of 
Unique r 
Varlanea 



Focantial 

Unique 

V^ldity 



A 
G 
D 
E 
F 



*197 
.325 
.199 
.226 
*251 



,701 
.689 
.611 
.758 
,626 



*585 
• 537 
,428 
.636 
.448 



*218 
,139 
.374 
.138 
,301 



.467 
.373 
.612 
.371 

.549 



See Iquaclon 1 for e^lanatlon of statistical tami. 
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